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Executive Summary 

This report describes the state of practices of design reviews at NASA and 
research into what can be done to improve peer review practices. There are many types 
of reviews at NASA: required and not, formalized and informal, programmatic and 
technical. Standing project formal reviews such as the Preliminary Design Review and 
Critical Design Review are a required part of every project and mission development. 
However, the technical, engineering peer reviews that support teams’ work on such 
projects are informal, some times ad hoc, and inconsistent across the organization. The 
goal of this work is to identify best practices and lessons learned from NASA’s 
experience, supported by academic research and methodologies to ultimately improve the 
process. This research has determined that the organization , composition, scope , and 
approach of the reviews impact their success. Failure Modes and Effects Analysis 
(FMEA) can identify key areas of concern before or in the reviews. Product definition 
tools like the Project Priority Matrix, engineering-focused Customer Value Chain 
Analysis (CVCA), and project or system-based Quality Function Deployment (QFD) 
help prioritize resources in reviews. The use of information technology and structured 
design methodologies can strengthen the engineering peer review process to help NASA 
work towards error-proofing the design process. 
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1. Introduction 

1.1. Background 

Design reviews are a systematic way to manage the process of product 
development to ensure product design quality reflects and meets customer 
requirements within cost and time constraints. The Japanese Industrial standard 
JIS Z 8115-1981 defines design reviews as: 

Judgment and improvement of an item at the design phase, reviewing the 
design in terms of function, reliability, and other characteristics, with cost 
and delivery as constraints and with the participation of specialists in 
design, inspection, and implementation. 

There are two types of design reviews. Formal design reviews have standard 
policies and procedures. Each review is a key event in the process of product 
development and production planning. Informal design reviews are developed 
and conducted by individual reviewers. These reviews are used only as needed 
and their effectiveness can vary greatly. 

In a survey by the Design Review Committee of the Union of Japanese Scientists 
and Engineers (JUSE), few reported any actions to correct misunderstandings of 
design reviews. Some of the most frequently cited concerns were time and 
scheduling constraints, lack of staff experience, inadequate preparation, and 
shortfalls in communication, cooperation, and commitment. (Ichida 1996) 


1.2. Goal 

This report describes the state of practices of design reviews at NASA and 
research into what can be done to improve review practices. There are many 
types of reviews at NASA: required and not, formalized and informal, 
programmatic and technical. Standing project formal reviews such as the 
Preliminary Design Review (PDR) and Critical Design Review (CDR) are a 
required part of every project and mission development. However, the technical 
engineering peer reviews that support teams’ work on such projects are informal, 
ad hoc, and inconsistent across the organization. The goal of this work is to 
capture the state of peer review practices currently at NASA and to go beyond 
that and identify best practices and lessons learned from NASA’s experience, 
supported by academic research and methodologies to ultimately improve the 
process and work towards error-proofing the process. 


1.3. Method 
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Based out of NASA Ames Research Center in Moffett Field, CA, this design 
review research first referred to official NASA documentation and NASA mission 
webpages on formal design reviews as well as NASA interviews. Initial 
conversations w'ith engineers and managers at NASA Jet Propulsion Laboratory in 
Pasadena, CA indicated that we should concentrate on the informal engineering 
peer reviews. 

In additional to interviews at Ames Research Center, two visits were made to 
NASA Jet Propulsion Laboratory in Pasadena, CA on June 19-20, 2003 and July 
15, 2003. These conversations initiated dialogue with several officials at JPL, 
Ames, Langley, and Goddard via phone, e-mail, and in person and are detailed in 
the appendix. Observations were also made from the Kepler Ground Segment 
(pre-SRR) peer review held at NASA Ames Research Center on June 26, 2003. 
Finally, there were opportunities for discussion at the “Space Mission Challenges 
in Information Technology” conference in Pasadena, CA on June 13-17, 2003; the 
“NPI Roundtable on Reliability and Validation” at Stanford University on June 24, 
2003; and the “International Research Roundtable” at the Swiss Federal Institute 
of Technology in Lausanne (Ecole Polytechnique Federale de Lausanne) on 
September 11, 2003. 


2. NASA Review Process 
2.1. NASA Centers 

The National Aeronautics and Space Administration (NASA) is an agency in the 
U.S. federal government with the mission of conducting research and developing 
operational programs in the areas of space exploration, artificial satellites, and 
rocketry. The agency came into existence on October 1, 1958, and there are 
currently 11 facilities in the agency [13]: 

- NASA Headquarters - located in Washington, D.C., exercises 
management over the space flight centers, research centers, and other 
installations that constitute NASA. 

- Ames Research Center - specializes in research geared towards creating 
new knowledge and new technologies that span the spectrum of NASA 
interests. 

- Dryden Flight Research Center - innovates in aeronautics and space 
technology - the newest, fastest, the highest - as the lead for flight research. 

- Glenn Research Center - develops and transfers critical technologies that 
address national priorities through research, technology development, and 
systems development for safe and reliable aeronautics, aerospace, and 
space applications. 


5 



NASA Technical Report 


Technical Engineering Peer Reviews 

NASA 


- Goddard Space Flight Center - mission to expand knowledge on the 
Earth and its environment, the solar system, and the universe through 
observations from space. 

- Jet Propulsion Laboratory - managed by the California Institute of 
Technology is NASA's lead center for robotic exploration of the Solar 
System and mission design. 

- Johnson Space Center - continues to lead NASA's effort in Human Space 
Exploration, from the early Gemini, Apollo, and Sky Lab projects to 
today's Space Shuttle and International Space Station programs. 

- Kennedy Space Center - America's “Gateway to the Universe,” leading 
the world in preparing and launching missions around the Earth and 
beyond. 

- Langley Research Center - continues to forge new frontiers in aviation 
and space research for aerospace, atmospheric sciences, and technology 
commercialization to improve the way the world lives. 

- Marshall Space Flight Center - is world leader in the access to space and 
use of space for research and development to benefit humanity, bringing 
people to space and space to people. 

- Stennis Space Center - responsible for NASA's rocket propulsion testing 
and for partnering with industry to develop and implement remote sensing 
technology. 

This project was based in Ames Research Center but worked very closely with the 
Jet Propulsion Laboratory and supplemented by some conversations with 
individuals at Goddard and Langley. 


2.2. NASA Life Cycle 

For decades, the National Aeronautics and Space Administration has applied 
effective design principles with appropriate peer reviews and periodic systems 
design reviews to result in high reliability aerospace design . NASA has a well- 
defined life cycle which consists of the following areas, shown in Figure 1. 
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Figure 1 : NASA’s Defined Life Cycle 


Like many organizations, NASA uses these phases as a means to organize 
decision points, illustrated in Table 1. Requirements definition begins in Phase A, 
with refinements and baseline occurring in Phase B. Lower level requirements 
are derived between Phases B and C, and major requirement definition is 


6 




NASA Technical Report 


Technical Engineering Peer Reviews 

NASA 


completed for ail levels by Phase C. Design reviews are at key transition points 
along this life cycle 


T able 1 : NASA Life Cycle chart showing requirements and reviews 


PHASE: 

A 

B 

C 

D 

E 


Prel im man' 

Definition 

Design 

Develop nvni 

O perditions 


Analysis 





Activities: 

Concept n a 1 

Preliminary 

Detail design 

Final design 

Support 


si tidies 

design 

System 

development 

pDXjtKTt 


Exploration of 

C onccpt 

devc lop nieni 

Fabrication 

improNenkfnt 


tiller mimes 

solution 


Test 


Requirement 

Program 


Segment 



Related 

Pta 

Baseline 

Specs 

Maintain 

Maintain 

Documents: 


i System 


Specs 

| Specs 


Draii 

Specification 

Piivc 




In v stem 
.Specification 





Re views; 


SRR PDR 

CDR 

SAR FRR 

ORR 


All NASA missions and spacecraft are subject to a technical design review 
process. The primary objective of this program is to enhance the probability of 
success by identifying potential or actual design problems in a timely manner. 
There are a number of system reviews which are performed throughout the 
lifecycle, including: 

- System Concept Review (SCR) 

- System Requirements Review (SRR) 

- Preliminary Design Review' (PDR) 

- Critical Design Review (CDR) 

- Mission Operations Review (MOR) 

- Pre -Environmental Review (PER) 

- Pre-Shipment Review (PSR) 

- Systems Acceptance Review (SAR) 

- Flight Operations Review (FOR) 

- Flight Readiness Review (FRR) 

- Launch Readiness Review (LRR) 

- Operational Readiness Review (ORR) 

The Technical Design Review Program consists of a subset of such system 
reviews, depending on whether it is a spacecraft or an instrument, or new or 
follow-up mission. 


2.3. Types of Reviews 
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To help understand the organizational aspects of the design review system at 
NASA, we performed a Customer Value Chain Analysis (CVCA). CVCA is a 
design tool taught in Stanford University’s dfM course me317 
('http://me317.stanford.edu') that helps an organization to understand the value 
proposition for each stakeholder. It lists the pertinent parties involved in the 
product including stakeholders, customers, partners, and regulators and identifies 
the relationship and flow of money, materials, resources, complaints, and 
information among the parties. 

The CVCA in Figure 2 (the appendix contains breakdowns of the chain by 
category) shows the different stakeholders involved in typical projects from a 
design review perspective. NASA is a matrix organization which brings members 
from different functional line organizations together for projects. The different 
types of review teams that may impact a project are shown in the dotted circles. 
Formal review boards sit on system reviews such as the PDR and CDR. Peer 
reviewers are gathered from different organizations within NASA and even 
outside. As the CVCA shows, the funding for both these reviews comes from 
within the project and not from Headquarters or other NASA offices. 



Figure 2: Customer Value Chain Analysis (refer to appendix) 
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In addition to the system reviews, there are also externally motivated project 
reviews. The “Red Teams” are comprised of senior management representatives 
that periodically review the process; they are charged with evaluating the 
approaches used to manage and mitigate risks during the lifecycle and report to 
the System Management Office. The Independent Program Assessment Office 
(IPAO) conducts independent evaluations of NASA programs and projects to 
ensure that the technical and programmatic commitments are being met. 
Independent Review Teams (IRT) consist of highly knowledgeable specialists 
both internal and external to NASA and conduct reviews as requirements mandate. 


2.4. Formal Reviews 

In the NASA life cycle, two key reviews are the PDR and CDR. Figure 3 shows a 
representation of the JPL life cycle including major reviews. 
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Figure 3: JPL life cycle including major reviews 


The Preliminary Design Review (PDR) is the first major review of the detailed 
design and is normally held prior to the preparation of formal design drawings. 
PDR’s are conducted to confirm that the approach for the system's design is ready 
to proceed into the detailed design phase. A PDR is held when the design is 
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advanced sufficiently lo begin some testing and fabrication of design models. 
Detail designs are not expected at this time, but system engineering, resource 
allocations and design analyses are required to demonstrate compliance with 
requirements. 


The Critical Design Review (CDR) is held near the completion of an engineering 
model, if applicable, or the end of the breadboard development stage. This should 
be prior to any design freeze and before any significant fabrication activity begins. 
The CDR should represent a complete and comprehensive presentation of the 
entire design. CDR’s are conducted to demonstrate that the detailed design is 
complete and ready to proceed with coding, fabrication, assembly and integration 
efforts. 


2.4.1. Guides 

For formal reviews, there are a number of guides and documents to help projects 
through the review process. The JPL documentation PD-ED-1215 [C] outlines the 
practice of conducting technical reviews. Figure 4 outlines the generic process 
for implementing project reviews from the review board’s standpoint. 



Figure 4: Project review process overview 


Figure 5 illustrates the preparation for project reviews from the design 
presentation side. 
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Figure 5: Project review process: review preparation 


Some groups have design review forms with detailed activities. Below, Table 2 
lists review forms used for the PDR and CDR for software verification and 
validation. These contents are to be verified and remarks are to be noted. [1] 


Table 2: PDR and CDR Design Review Forms for Software V&V 
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Discrete quality 3nd adequacy checks have been 
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Table 3 below demonstrates a standard review checklist which is used as an aid to 
review planning. The activities are to be performed in the order listed by the 
people indicated. This checklist is very high-level and does not describe how to 
handle technical aspects. 


Table 3: Standard review checklist 


Review Activity 

Lead Person 

Generate project review plan 

Project manager 

Establish and document charter 

Convening authority 

Establish and document scope, objectives, success criteria, 
and prerequisites 

Responsible individual 

Select review board 

Convening authority, 
responsible individual 

Announce schedule and agenda 

Responsible individual 

Prepare for review: 

• Schedule conference room 

• Arrange for audiovisual equipment and support, 
refreshments 

• Identify presentation team 

• Develop presentation guidelines 

• Hold presenters’ meeting 

• Assemble material to be reviewed, and distribute material to 
board 

• Generate presentation and backup material 

• Dry run or story board presentation 

• Update, produce, and print presentation material, and 
distribute it to the board 

• Prepare slide and transparencies, and distribute them to the 
presenters 

Responsible individual 

Study material prior to review 

Board members 

Conduct review 

Board chair 

Conduct post-review meeting: 

• Identify key findings and recommendations 

• Develop board consensus 

• Draft board report 

Board chair 

Consolidate and filter recommendations for actions (RFAs) 

Board chair, responsible 
individual 

Accept RFAs as action items, advisories, or rejected items 
• Identify critical action items 

Responsible individual 

Complete and issue final board report 

Board chair 

Submit metrics to the Office of Engineering and Mission 
Assurance 

Responsible individual 

Prepare and issue RFA disposition plan 

Responsible individual 

Approve disposition plan 

Convening authority 

Approve action item closures 

Responsible individual 

Review action item closures; provide feedback to responsible 

Board chair, selected board 

| individual and convening authority 

members [ 
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2.5. Review Examples 

2.5.1. ASIC Program 

In the ASIC program at NASA, the review process and other assessments are an 
important part of the program. The user and designer participate together in 
review early in the process. Review board members include individuals from 
design, product architecture, customer engineering, CAE/CAT, ASIC center 
personnel, resident ASIC experts, parts reliability, quality assurance, procurement, 
and ASIC vendors. General activities include verifying requirements, identifying 
problems, locating causes of problems, addressing concerns, making 
recommendations, and developing communication channels. 

These include reviews for specifications and requirements, implementation 
(schematics), preliminary design, critical design, and chip sign-off (build- 
readiness/flight build). The specification review includes checks the 
completeness of specifications and the compatibility of existing design work and 
future applications. The implementation review looks at the specification 
implementation. The PDR reviews parts specification, verification of reports, test 
summary, package information, and schematics and directory structure. The CDR 
reviews part specification and includes a design verification check. 


2.5.2. International Space Station Fluids and Combustion Facility 

The Fluids and Combustion Facility (FCF) is a permanent, modular, multi-user 
facility to accommodate microgravity space experiments. Even with the cost of 
FCF development included, experimentation using FCF on the space station will 
cost only half of what it did on the space shuttles. 

The Preliminary Design Review for the FCF took five days, including sessions on: 

- Day 1: FCF System Preliminary Design Review 

- Day 2: FCF Software, Common Subsystems, and System Summary; CIR 
Delta-PDR 

- Day 3: FIR Preliminary Design Review 

- Day 4: SAR Conceptual Design Review 

- Day 5: FCF PDR Executive Session 

The review teams included members from systems, structures, thermal, avionics, 
software, IOI, S&MA, combustion, fluids, management. The design review 
teams presented requirements, overviews, parts and features, and hardware lists. 
Other documents included flight drawing tree, FMEA, test plan, software 
requirements documents, reliability reports, compliance matrices, risk 
management plan, standards list, acceptance plan, and mechanical drawings and 
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schematics. Table 4 lists the documentation preparation plan for the FCF’s PDR, 
including identifying the authors, responsibles, and key dates. 


Table 4: Documentation preparation plan for the FCF PDR 
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From this, the review board produced memorandums like checklists, minutes of 
attendees, summary of issues discussed, action items, as well as other 
recommendations and conclusions. Table 5 lists a design review checklist for the 
CER Gas Chromatograph which includes the areas and items reviewed as well as 
comments. 


14 












NAS A Technical Report 


Technical Engineering Peer Reviews 

NASA 


T able 5: Design review checklist 
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2.5.3. Other Missions 

New Horizons is the first mission to Pluto, its moon, Charon, and the Kuiper Belt 
of rocky, icy objects beyond. Its Preliminary Design Review lasted 3-days at 
Applied Physics Library in Laurel, MD. The 10-member review panel of 
spacecraft and system engineering experts from APL, NASA JPL, Goddard, and 
Southwest Research Institute examined New Horizons' mission plans and 
spacecraft design, with APL Space Department's chief engineer chairing. 

The Stratospheric Observatory for Infrared Astronomy (SOFIA) is a joint 
effort between NASA and the German Aerospace Center, DLR. NASA’s prime 
contractor was the Universities Space Research Association (USRA), which 
provides a mechanism through which universities, the government, and other 
organizations can further space science and technology. SOFIA’S 4-day Critical 
Design Review took place in Waco, Texas, where USRA subcontractor Raytheon 
is modifying the aircraft to house the telescope. The event bridged design and 
manufacturing stages, where a successful review meant that the design is 
validated and will meet its requirements, is backed up with solid analysis and 
documentation, and has been proven to be safe. The industry team led by USRA 
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presented the complete system design developed to make sure that technical 
issues have been properly addressed. SOFIA'S CDR completion granted USRA 
permission to begin manufacturing of hardware. 

The critical design review for the Mars Surveyor Orbiter Color Imager 
(MARCI) and Mars Surveyor Lander Descent Imager (MARDI) took place on 
one day, lasting from 8:30am to 5pm. In it, the chairman of the review led the 
discussion, prepares the official report of the results, and is in charge of 
developing the system to operate future Mars missions. The lead engineer for the 
new cameras presented most of the technical details. Members of the review 
board were a JPL engineer in charge of science instruments for the Pathfinder, a 
SDSU astronomer who built and uses cameras on telescopes, and the designer of 
the Mars Observer and Mars Global Surveyor cameras. 

The Stardust mission will gather samples of dust as it flies by a comet and return 
them to Earth. The PDR had an independent review board appointed by the space 
agency, and marked the end of the mission's concept definition phase (Phase B) 
and the start of design, development and fabrication (Phases C and D). The CDR 
confirmed that the design is complete and subsystems are on schedule for 
spacecraft integration. 

The Lunar Orbiter missions were five missions that were launched with the 
purpose of mapping the lunar surface before the Apollo landings. Each landing 
was made successfully and, in total, they were able to photograph 99% of the 
moon. The PDR was conducted by Boeing and NASA. It checked any specific 
technical area or major subsystem before a final decision was made to freeze the 
design. The CDR concentrated on the components and subsystems to see if they 
passed as acceptable for fabrication and testing; if approved, changes were held to 
a minimum. Various other reviews took place during fabrication and a formal 
acceptance review was conducted at the completion point 

Cassini-Huygens was launched in 1997 to reach Saturn by 2004. The mission is 
composed of two elements. The Cassini orbiter, built and managed by JPL, will 
orbit Saturn and its moons for four years. The Huygens probe, built by the 
European Space Agency, will dive into the murky atmosphere of Titan and land 
on its surface. The reviews consisted of JPL and other NASA and independent 
reviewers, supported by the European Space Agency and the Italian space agency 
Agenzia Spatiale Italiana. 


2.6. Lessons Learned 

2.6.1. NASA Programs 

The successes and failures of NASA missions have also provided lessons learned 
for the organization’s design review practices, listed in Table 6. 
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T able 6: Lessons learned from some NASA missions 


Mission 

Event 

Lesson Learned 

Huygens 

NASA personnel were denied 
access to the designs for the 
spacecraft by partners, 
making resolution of 
problems difficult or even 
impossible 

Important to retain engineering rights to 
all designs, analyses, procedures, and 
test results 

Skylab 

Fell to earth in showering 
debris over uninhabited parts 
of Australia and the Indian 
Ocean. 

Specific design reviews which are based 
upon an analysis of drawings can 
inadvertently overlook important features 
such as operational compatibilities 

Mars Climate 
Orbiter 

Navigation errors 

Inadequate reviews missed use of 
different units, key personnel were 
missing from critical design reviews 

Mars Polar 
Lander 

Premature shutdown 
scenario. The spacecraft was 
not designed to send 
telemetry during descent. 

Investigation was hampered by lack of 
data. The decision not to send telemetry 
during descent was severely criticized by 
review boards yet still not changed. 


A number of studies have been done throughout NASA to improve the review 
process. The ASIC program [3] identified the following considerations: 

- Plan for reviews at the beginning of a program. 

- Don't underestimate the importance of selecting appropriate board 
members. Remember you will rely on their expertise to achieve first-pass 
silicon. 

- Identify the participants for reviews early enough so that they may receive 
all necessary review material in time for their analyses. 

- Work with the vendor's review methodology to reconcile your 
organization's goals for a particular review with those of the vendor. 

- Build the reviews into the contract and the statement of work so that 
sufficient resources will be available from the vendor to properly support 
the reviews and action items generated from them. 

A survey of software Validation & Verification processes and methods at NASA 
[1] identified the following recommendations: 

- Organize modeling teams with responsibility for entire sub-systems to 
ensure internal coherence and communication. 

- Evaluate testing coverage of autonomous software 

- Develop tools to mitigate the effect of late changes to requirements. 

- Develop better model validation processes and tools 

- Use new graphical tools to provide visual inspection and modification of 
mission profiles, as well as constraint checking 
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Develop tools and simplify the modeling languages so spacecraft experts 
can encode models themselves and explain the models to test engineers 


more effectively. 

Simplify the specification of goals and automate consistency checking 


2.6.2. Lessons Learned Information System 


The Lessons Learned Information System (http://llis.nasa.gov/llis/llis/llis.html) is 
a reference database of different lessons from NASA projects. It is provided to 
NASA personnel and approved NASA contractors. Searching for “design 
review” in the system provided 154 different lessons, summarized in Table 7. 
Some of the lessons of interest pertain to process considerations and guides, but 
many only point out specific items that should be considered in future reviews of 
a similar system. 


T able 7: Lessons learned on “design review” in LLIS 


Type 

LLIS Database Entry 

Description 

Coverage 

0017, 0564, 0634, 0637, 0638, 
0640, 0644, 0740, 0763, 0869, 
0885, 0886, 0905, 0906, 0908, 
0916, 0923, 0932, 0981, 0989, 
1120, 1184, 1185, 1200 

Specific items to review, such as 
certain analyses to perform in future 
reviews 

Considerations 

0271 , 0286, 0387, 0393, 0440, 
0584, 0588, 0862, 0917, 0974, 
1063, 1089,1180,1196,1278 

More general considerations for 
reviews, such as when or what to 
include 

Guides 

0648, 0655 (PDR), 0656 
(HR/CR), 0657 (CDR), 0667, 
0668 (PSR), 0681, 0682, 0728, 
0761, 0786, 0789 (SIR), 0929, 
1211 

Review method guides and guidelines 

Tools 

0371 (FMECA), 0599 
(Taguchi), 0733 (PFR), 0738, 
0791 , 0825 

Design tools to use during reviews and 
items to support reviews, like databases 
and forms 

Organization 

0495, 0533, 0534, 0619 

Review authority and personnel 
inclusion 

Benefits 

0582, 1276 

General reasons to do reviews 


2.7. Issues 

Initial surveys and interviews showed high confidence in the formal review 
process at NASA. The general consensus was that with the right people all 
problems should be caught in design reviews. Though there is high confidence in 
the individuals at NASA, there can be great variation in how design reviews are 
executed, depending on the combination of individuals involved. In addition, 
even though there are many guidelines there are some times inconsistencies in 
implementation. For some very small missions, a Single Design Review (SDR) is 
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done, combining the Preliminary and Critical Design Reviews. Engineers at 
Goddard who have been through this have said that it is a bad idea. 

NASA has taken a closer look at how design reviews communicate information 
between reviewers and designers. Bill Parsons, space shuttle program manager, 
said he hopes to open the doors of communications between NASA's four 
spaceflight centers. NASA engineers debated the potential damage to Columbia 
right up until landing day but never notified NASA's top management. "Maybe 
we have not shown people how they need to get in the loop of the formal 
decision-making process," Parsons said. "We're going to go and characterize that 
for people and see how they may do that better." [8] 


A major issue is with how designers view reviews. One project manager even 
said that reviews can be “dangerous” in that the project might assume that the 
review can catch everything for them. Some managers strongly believed that 
design reviews can and should catch any problem. Others feel that the design 
review is a weak process which is completely dependent on the individuals 
involved. And there are even others that feel the complexity of the systems has 
exceeded our abilities to grasp it. 


In some interviews and conversations, discussion of design reviews begin with a 
statement along the lines that “99.9% of the value of reviews is preparing for 
them.” This statement refers largely to the formal reviews which are a required 
part of the NASA life cycle. The informal peer reviews are viewed differently as 
part of the preparation process. Though important, these reviews lacked the 
consistency and formalism of the system reviews. 


3. Peer Reviews 

3.1. Background 

Peer reviews were first introduced around 1690 and have served as a useful tool in 
evaluation in science and engineering. Though the process is not perfect, many 
believe it is far better than alternatives. (O’Reilly 2002) There are a number of 
models of summative evaluations. Eibeck (1996) lists the three prototypical 
models as: 

1 . Review 

2. Checklists 

3. Experimental/user observation 

Trenner (1995) says the advantage of peer review comes when the designer is 
inexperienced or experienced developers are “too close” to their work to see it 
objectively. Difficulties can come in peer reviews when designers feel threatened 


19 



NASA Technical Report 


Technical Engineering Peer Reviews 

NASA 


or demoralized, a reluctance to criticize, a pooling of ignorance, or the review 
doesn't show the severity of the problems identified. Peer reviews can also fall 
into the trap of being a cosmetic exercise with no commitment to making the 
changes suggested. 

These peer reviews are key to success in the formal review. However, in 
initiating conversations with engineers and manager across NASA about “peer 
reviews,” it was clear that the term meant different things to different people. 


3.1.1. Official Definitions 

Jet Propulsion Laboratory’s Project Review document [G] defines peer reviews as: 

A peer review is a working-level, in-depth review convened as needed to 
evaluate an analysis, concept, design, product, or process thoroughly. 

Peer reviews focus on early detection of flaws. They are also used to 
detect flaws in engineering products and processes prior to delivery from 
development organizations to test and operations organizations. Special 
in-depth reviews called pre-reviews are conducted before higher level 
design reviews. 

Goddard’s System Management Office [4] describes Engineering Peer Reviews 
(EPR’s) as a resource for product design teams (PDT’s) to “identify potential 
engineering design and implementation flaws, and increase the probability of 
success.” These EPR’s address: 

- Requirements and Resource Adequacy 

- Systems Management Processes 

- Design Adequacy: Drawings, Schematics, Analyses, Parts and Materials 

- Compliance with Policies, Procedures, Standards and Best Practices 

- Implementation Adequacy 

- Manufacturing Processes 

- Verification Approach: Tests, Analyses, Simulation 

- Verification Results: Data Adequacy, Observed Margins, Trends, 
Anomalies 

- Claims of heritage from previous missions 

- Lessons Learned (applied and learned) 


3.1.2. Interview Scope 

To facilitate discussions with NASA personnel, this research used a definition and 
scope of peer reviews for these discussions as informal, in-depth technical 
reviews, usually held before major reviews like PDR and CDR as pre-reviews. 

The number of people involved can include anywhere from a conference room 
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full to just one or two reviewers in an engineer’s office. Even the definition of a 
“peer” varied somewhat. Peers are usually other people from a similar technical 
background, though some managers emphasized that peers should also be peers in 
term of organizational hierarchy, i.e. no managers or other “bosses.” 


3.2. Documentation 

There are a number of guidelines, checklists, and documentation for system 
reviews like the CDR, but guides for peer reviews are quite limited. Often, peer 
reviews are simply mentioned as pre -reviews for the system reviews. 

JPL D-11381 [F] simply recommends “a series of detailed peer reviews be 
conducted prior to the Preliminary Design Review (PDR) and, especially, the 
Critical Design Review (CDR).” The peer reviews should be informal but 
structured, including a checklist, and the team should summarize the review at the 
formal PDR or CDR. 

The NASA Mission Design Process [B] recommends peer reviews to be 
conducted periodically through Phase A (Mission Analysis). The group should be 
composed of individuals chosen from outside the project. Review of analyses, 
drawings, and other design documentation versus viewgraphs is recommended. 

In Phase B (Definition/System Design), the technical part of the review should 
also examine associated cost and schedule data. 

Perhaps the most extensive documentation on peer reviews comes in D-10401 [G], 
It calls for peer reviews to be “convened, as needed, as working-level reviews to 
evaluate detailed aspects of a product or process.” Though it explicitly states that 
peer reviews are to be informal and not subject to the formal project review 
requirements, it suggests the following guidelines. 

- Peer reviewers should not be currently working in the project/major task 
element being reviewed. 

- The plan for the use of peer reviews, and the plan for recording their 
results, should be included in the review plan. The number and subject 
areas for peer reviews should be generally scoped by the project manager 
or designee, in order to provide sufficient budget to implement them 

effectively. 

- A record, consisting of the purpose and date(s) of all peer reviews 
conducted should be maintained by the project as part of the Product 
Delivery’ System records. Further, the records should include, for each 
review conducted, a list of reviewers, findings, recommendations, and 
action items accepted by the project. 

- Peer reviews should thoroughly examine product or process issues. Peer 
reviews are also useful to detect and correct deficiencies in engineering 
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products and processes prior to delivery from development organizations 
to test and operations organizations. 

- Peer reviews should be held during development of products or processes. 
They can be held at any time, but are particularly useful prior to higher- 
level design reviews. Peer reviews are limited to a single product or 
process to ensure a thorough, in-depth evaluation. 

- Peer reviews can be called by the project or by the cognizant line 
organization (Section Manager, Group Supervisor, etc.), when a concern 
exists. The concent could be effectiveness of the ongoing review and 
oversight of the effort in some area of development - either the design 
itself or the design status. It also could be that a known or suspected 
problem in the design needs investigation. 

Even with these guidelines, it still emphasizes that reviews should be “kept simple 
and informal to minimize the cost and effort. Likewise, the supporting 
documentation for a peer review should be kept simple and informal.” 


3.3. Method 

Peer reviews are usually held on the sub-system level, though reviews for specific 
components are held in support of the sub-systems as well. Ideally, peer reviews 
would be done on every subsystem and component in the greatest of detail with 
all the foremost experts, but that is not always possible. The project manager 
usually works with the sub-system section leaders to discuss how much time and 
resources should be invested in each sub-system review. 

It is up to the program manager to decide who to include in reviews. Currently 
there is no organized system or list of reviewers in place for managers to refer to. 
The process is quite informal but intuitive in many ways. The sub-system leader 
will usually look for reviewers in his or her own line organization, so often will 
just choose his or her colleagues or ask the line manager for suggestions on 
reviewers. The main constraint is to choose people who are not working on the 
same project. For example, if the subsystem is in electronics, reviewers should be 
people who are in electronics, like former cognizant engineers. When experts are 
needed from other areas, often the best place to start is a manager of that section 
or line organization. The peer reviews are paid for by the project where the 
reviewers can bill their time to the project. Many times, however, the reviewers 
simply volunteer their time to help out their peers. 

The peer review method varies from project manager to project manager. One 
project manager had a formalized method which for his “engineering peer 
meetings.” As pre -reviews, his rule of thumb is to conduct the peer reviews about 
a week or 10 days before the formal review. This gives enough time to react to 
suggestions and criticisms. To do a review too early, even as much as a month, 
likely means the review of something that is not the true status by the time the 
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formal review comes. However, ihe planning of these peer discussions must start 
to take place well in advance. It can take up to six months to get a good handle of 
a medium size mission. It takes a month to figure out good questions to ask, a 
month to collect questions about similar subsystems in the past, a month to find 
out who to talk to, two months to conduct the interviews, and another month to 
put the data into something that’s reasonable and use probabilistic risk tools. 

Some project managers are involved in the process by preparing templates of 
types of questions that reviewers should be concerned about, and are generated 
from historical and lessons learned sites. Most reviewers don’t prepare anything 
before there reviews. 


Langley Management System’s LMS-CP-5508 [E] outlines the procedure for Ad 
Hoc Technical Reviews in order to increase the probability of success of 
Langley’s programs and projects through technical reviews, shown in Figure 6. 
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Figure 6: 


Representation of Ad-Hoc Technical Reviews at Langley 
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To get an idea of the typical size of these reviews, according to one branch chief, 
peer reviews are usually fairly short, on the order of around 2 hours, and 
participants are usually notified anywhere from one day to one week in advance. 
The project manager typically decides who can be of most benefit and invites 
them, usually holding the number to around 4-8 people. Branch management is 
typically invited but not required. 


3.4. Issues 

There is strong sentiment at NASA that peer reviews give the most benefit and 
cost the least amount of time and effort in the life cycle. Because these peer 
reviews are more or less “voluntary,” they can be done with more flexibility and 
cover the topics that are important to the designers not the reviewers. At the same 
time though, they lack consistency in implementation. A good project peer 
review is very dependent on strong leadership and involvement by the project 
manager. 

While standing reviews like the PDR and CDR are highly structured and 
formalized, the technical peer reviews that are an important pre-review for these 
programmatic reviews are not. It is in these informal reviews that the engineers 
and managers must work out the details that can be missed in formal reviews. 

The key is success in these peer reviews though is a balance between discipline 
and freedom. 


4. Design Review Analysis 

From the interviews with NASA engineers and managers and comparing them to 
research done in existing literature, NASA must further consider key five areas in 
-working towards improving design reviews: the role of design reviews in the 
organizational structure, the composition of the design review boards, the scope 
of analysis for each peer review meeting, the personal approach taken by the 
team and reviewers, and the use of information technology and other tools to aid 
the reviews. 

4.1. Organization 

4.1.1. NASA lessons 

At NASA JPL, the peer reviewers are usually full-time engineers and managers 
who donate some of their time to help review projects for their colleagues. 
However, at NASA Goddard, there actually is a full time review group which 
does the peer reviews. Some independent review boards are used at NASA, 
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though they are often there more to report on programmatic issues to 
Headquarters. At NASA Ames, there was extensive contracted use of 
independent reviews on their projects. 

Most JPL personnel expressed a belief that reviewers should be full-time 
engineers and managers so that they don’t lose their technical expertise and can 
stay current with the state-of-the-art at NASA. However, some expressed some 
interest in the idea of having a part-time review program where engineers could 
work as a “full-time reviewer,” perhaps for a period of several months. This 
rotating review board could train the reviewers and allow them to review for 
several months then return to their original position, allowing them to stay 
“current.” 


At Ames, managers said they’ve learned that it was clear that it was necessary to 
budget for reviews, particularly the PDR and CDR. The requirements were levied 
in the contract. Independent and peer review's weren’t really budgeted for initially. 
Though some time is allocated for peer reviews, for the most part, money isn’t 
allocated and when the budget becomes tight, it is not uncommon to cut peer 
reviews. Some managers even said the way to make the peer review system ideal 
is to make the personnel available to the project at no cost. 


4.1.2. Literature review 

Ichida et al. (1996) examined possible relationships between the design review 
board and the designers. They identified three models. The first has design 
reviews dominated by the design department. The advantages are that reviews 
can be introduced with minimal resistance and the designers can better prevent 
infringements upon their authority. The disadvantages are that the designers will 
more likely guard their own turf when determining action items, scheduling, and 
follow-up. Having a design review office advising the designers can result in 
more impartial action items and subjects to be addressed, but there can be 
difficulty in gaining sufficient understanding from both sides on the design and 
the philosophy of the design review. Finally, there can be a shared relationship 
where the design review board is elevated to a joint-decision making role within 
the design team. This gives the benefit of an impartial approach and suited 
towards phase-specific (like gate) management, however with even less sense of 
ownership, the designers would feel a loss of authority and easily overlook 
routine activities. 

Though not design reviews in the strictest sense, many organizations use a type of 
stage-gate process to guide their product development. ABB is a multi-national 
organization with two product divisions (power and automation technologies) that 
also serve external channel partners, Financial Services, and Group Processes. 
Similar to NASA’s Life-cycle model, ABB implemented a business decision 
model supported by the Gate Model, detailed in Table 8. 
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T able 8: The ABB Gate Model 


Gate 

Title 

Description 

GO 

Stan project 

Initiates the feasibility evaluation. The focus is on analysis of the 
requirements. 

G1 

Start planning 

Defines the scope of the project. The requirements agreed here will 
control the planning. 

G2 

Start execution 

Marks the agreement on requirements, concept, and project plan. The 
focus is on specification of functions and architecture. 

G3 

Confirm 

execution 

Confirmation that target dates can be met and that the project executes 
according to the project description and plan. After this gate, the 
focus is on implementation. 

G4 

' Start 
introduction 

Release for acceptance testing. Focus is on validation on preparation 
for the market introduction and on production preparations. 

G5 

Release 

product 

Handover of the results to the line organization. Indicates also that 
the project activities should be finished and focus is on finalizing any 
remaining issues. 

G6 

Close project 

The project ends. 

G7 

Retrospective 

investigation 

A follow-up of the project to check if the results are satisfactory and 
feedback experiences to the organization. 


The model serves as a framework for various activities, included in product 
development. Each organization runs their process through the Gate Model 
independently with a Gate Owner who has the authority to start/stop, fund, and 
staff the project, including the Gate Assessor. The Gate Owner is usually a 
manager responsible for the business that the project is focusing on. The Gate 
Assessor is an individual not directly involved in the project who is responsible to 
perform the assessment on behalf of the Gate Owner. The project leader aids the 
Gate Assessor in the assessments. Though there is no single organization which 
maintains and runs the Gate Model, ABB created a group- wide process 
organization to take ownership of the improvement activities and ensure 
consistent implementation of process standards across the company. [141 

At Genera] Electric, the design review is integrated into the design process 
through interaction between individual contributors, key technical specialists, and 
the Chief Engineer’s Office. By initiating informal meetings with Chief 
Consulting Engineers and the Chief Engineer throughout the design process, the 
designers can exchange data and discuss issues and ideas. Similar to NASA, 
these informal consultations are excellent preparation for formal design reviews 
and may consist of various meetings and discussions, including senior/staff 
engineers, peer engineers, technical specialists, or Chief Engineer’s Office 
representatives. These design reviews are not tollgate reviews, program technical 
reviews, nor scorecard reviews. 

There are other review models that can be followed. In the field of civil 
engineering in Germany, construction firms use full-time reviewers who are paid 
to review their projects with a share of the contract. In turn, however, these 
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reviewers assume half the liability of the project should anything go wrong. The 
U.S. government has had some programs where after contractors bid for a project, 
the runner-up contractor is hired as the reviewer for the winning contractor, as 
both are familiar with the technical details of the project. All contractors who bid 
for the job are made aware of this arrangement from the beginning. 


4.1.3. Discussion 

Though most interviewees did not have suggestions on major changes to the 
review process, when asked what they would do “infinite time and resources,” 
their answer was simply “get more reviewers.” Since the projects have to pay for 
their own reviews, it can dilute their effectiveness as there are a lot of competing 
interests within the project for the money. It is not uncommon for some sub- 
systems to not even do peer reviews. Even if the entire system is reviewed, there 
are no guarantees on its depth, consistency, or quality. It is not just getting a 
person, but getting enough quality time with them to comprehensively review the 
project. It is usually a privilege to even get two hours of some experts’ time to 
review drawings. Some times, two days are really needed to penetrate the design. 

Even though formal reviews are mandated by NASA Headquarters while peer 
review's are done on a voluntary basis for the projects’ own benefit, both are 
funded from the project and not the organization. For NASA to emphasize the 
importance of these reviews, it would make sense for them to guarantee funding 
perhaps from a separate source, at least on the formal reviews. There wall be 
some changes in the NASA accounting structure, and it may be able to put 
reviews under a corporate charge in the future. 

A review office can be useful to help project managers assemble and guide their 
review teams. Many showed interest in the idea of a rotating review' program 
where engineers work as full-time reviewers for several months at a time. For 
most informal reviews, that probably is necessary. At the least, the review office 
can help provide training material to reviewers and managers, including providing 
tips on conducting successful reviews. A review office can provide more than 
just technical reviewers. Other ways the review office can benefit the review 
process is through full-time review historians, cost/risk analysts, and system 
administrators for review databases and lockers. 

The review office could also maintain a list of reviewers for project managers to 
refer to. Though there was some discussion of rating reviewers and the usefulness 
of that, many felt the politics associated with it would outweigh the benefits. 
However, a completely objective list with only the reviewers’ names, contact 
information, and “resume” of their current position and background, as well as 
portfolio of projects that they have reviewed could be helpful. If this list could be 
cross-linked with project information, managers could look for certain similar 
projects to find qualified reviewers. 
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4.2. Composition 

4.2.1. NASA lessons 

Most NASA engineers and managers said they had no hard and fast rules on 
composition of their peer review boards other than to find the qualified people 
with the requisite skill set and experience. Usually these are the people they 
know from previous reviews or identified from talking to the managers in the 
appropriate line organizations. 

Some engineers and even project managers did believe that it is important to have 
only peers, not only in the sense of technical expertise, but also peers in term of 
organizational hierarchy present at peer reviews. By having no bosses or anybody 
who would “review the individual,” rather than just the project, it allows the 
engineers to be more open about challenges and problems they are facing. This 
allows the review to be conducted in a more informal manner. 

There have been differing views on the number of reviewers that should be 
present at a peer review. One school of thought is that the more reviewers, the 
better, as there are more “sets of eyes” to spot problems. However, one project 
manager felt very strongly that smaller sets of reviewers are better as they allow 
more personal interaction. In addition, a smaller peer review can be held in an 
engineer’s office where he or she has full access to any materials needed and 
allows them, for example, to look over a piece of paper and make marks directly 
on it together. This project manager said in his experience, the best “peer 
meetings” are with l-on-2 or 2-on-2 reviews, with 2 reviewers to one or two team 
members. In addition, he has the reviewers talk with him informally afterwards to 
discuss next steps. If there are 3 or more reviewers, then not only does it limit the 
dialogue, but it is necessary to make copies of the papers and makes the 
presentation process more formal. With only a few people, the team can just look 
over the same sheet of paper and have a more intimate dialogue. 


4.2.2. Literature review 

Research has found some general, though some times conflicting, rules of thumbs 
which can be considered. On the inclusion of supervisors, one view is that the 
review is highly influenced by the project supervisor whose responsibilities and 
knowledge of the project can more easily bring others to his or her point of view. 
According to this train of thought, the supervisor’s participation is therefore 
obviously needed. However, Freedman and Weinberg (1990) state very flatly that 
“Nobody should be present at a review whose role might create a conflict of 
interest.” Because team leaders can not be objective when leading a review of 
their work, they should never lead reviews of their own team’s work. 
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Regarding the ideal size of reviewing teams, D’astous, Robillard, et al (2001) 
found that it is subject to debate. Weller (1994) found that four reviewers are 
twice as efficient as three. Buck (1981) found no difference in the efficiency in 
two, thee, and four reviewers. Porter and Johnson (1997) determined size doesn’t 
influence anomaly detection rate. 


4.2.3. Discussion 


The key in design review' board composition is to have a group with the requisite 
skill set and right personality match to make the process run smoothly. More is 
not necessarily better. Both engineers and managers expressed the belief that 
having supervisors at peer reviews tended to hinder the process. However, not 
having managers physically present at the review's doesn’t mean that they should 
not be involved. The project managers and section leaders must take an active 
role in choosing the review board members, ensuring that the review will cover 
the technical aspects in a manner they find appropriate, perhaps generating a list 
of suggested topics to cover. 
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involved. Some people do require tougher criticism than others. It depends on 
the project manager to assemble a review team that can work well with the project 
team. No matter the style of the reviewer, it is important to emphasize the point 
of reviews is to identify opportunities for improvement. Managers should choose 
the members who will cover the issues they want. Formalism in the process can 
help keep things professional. 


Also, because these reviews are informal doesn’t mean there are no rules or 
documentation. The idea is to give the engineers and reviewers more freedom in 
covering the issues they want in a manner they are comfortable with. It is still 
necessary to report to the rest of the group the status of their piece of the system 
so that others can be informed of issues that may impact them. 


4.3. Scope 

4.3.1. NASA lessons 

Since projects have limited resources and competing interests, projects often 
cannot cover the breadth of all topics in sufficient depth. Tradeoffs must 
continually be made by projects as design reviews are one part of each project 
which must compete for attention in terms of both time and funds. It is not 
uncommon for some sub-systems to not even do peer reviews. 
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There are a number of documents which guide reviews. Langley’s LMS-CP- 
5505 [D] has all of the criteria and metrics associated with what is reviewed and 
what is needed to be successful based on which type of review is being held, i.e. 
PDR, CDR, etc. These same criteria and metrics are used for the particular major 
review. 

Even when peer reviews are done on all subsystems, they are not always done in a 
consistent fashion. Reviews are often too shallow technically. A number of 
factors must be considered when deciding whether a subsystem needs more 
resources and reviewers. Some of the criteria used to decide what is peer 
reviewed include: 


1. Complexity of the subsystem 

2. Technological Readiness Level of the subsystem 

3. Criticality of the subsystem to overall performance of the total system 

4. Breadth of the technical area over the entire project such as software 

5. Lack of sufficient information on the subsystem at a previous review (i.e. 
at a PDR, the subsystem was not to that level of maturity ) 

6. Past history of trouble in that area or subsystem. 
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agreement between the Review Chairperson and the Project Manager. 


4.3.2, Literature review 

Domer (1996) has researched and identified the logic of failure in complex 
situations. Humans have many inadequacies in dealing with complex systems. 
Complexity, dynamics, intransparence, and incomplete understanding of systems 
are all factors in recognizing and avoiding errors. Humans tend to badly 
mishandle temporal developments. They tend to extrapolate linearly and cannot 
handle exponential changes. The “slowness” of human thought often obliges 
shortcuts and prompts use of scarce resources as efficiently as possible. But 
instead of clarifying the complex relationships, humans often select one variable 
as central and economize around that. 

The scope of the review is important from both the reviewer and reviewee side. 
For the design team presenting, it is important that they provide a full picture of 
the situation. According to Leveson (2000), cognitive psychologists have 
determined that people tend to ignore information during problem solving when it 
is not represented. An incomplete representation actually impaired performance 
because they assume it as comprehensive and truthful. An incomplete problem 
representation can actually lead to worse performance than having no 
representation at all. One possible explanation for these results is that some 
problem solvers did worse because they were unaware of important omitted 
information. However, both novices and experts failed to use information left out 


30 



NASA Technical Report 


Technical Engineering Peer Reviews 

NASA 


of the diagrams which they were presented, even though the experts could be 
expected to be aware of this information. 

There currently is a dearth of proactive design tools to guide reviewers. The main 
aids available are questionnaire and checklists. However, it is difficult to create 
general and re-usable guides such as these. Eibeck (1996) studied different 
models for evaluations and found from feedback that detailed questionnaires were 
not successful tools for peer reviews. Review is highly subjective and can give 
significant information to the design team. A checklist can only address issues on 
a superficial level. 

Trenner (1995) says another difficulty of peer review comes in that there is often 
no system to quantify and compare the severity of the problems identified. 
Weyuker (1999) developed a questionnaire that computes a risk metric based on 
fifty-four of the most commonly occurring and severe problems in project 
management, requirements, and performance. There was a total of 440 points 
possible if every one of the problems were present, where a score less than 75 is a 
low risk and a score greater than 300 is a high risk. 


4*3*3* Discussion 

There are a number of structured methods to identify priority areas to cover in the 
peer review. It is necessary to do these tradeoff analyses because even if the 
design and review team says they will cover “everything,” chances are there will 
either be a few areas skipped or skimmed over. Often times, engineers are afraid 
that the planning of them takes too much time or that they inhibit innovation. 
However, as Barkan (1993) rebutted, planning not only results in significant 
reduction in development time and also leads to better quality. Using the right 
methods encourages innovation because they reduce the risk of failure, minimize 
oversights, and focus innovation on what counts. 


Project Level 

System Review QFD allows the project manager to analyze the elements of a 
system when trade-offs are necessary in making plans for peer reviews. The 
vision behind this QFD is to identify important areas to review in a system by 
quantifying their impact on the requirements for the project or mission. Doing so 
ensures the efforts spent during the project work towards that goal. By 
performing the System Review QFD before the engineering peer reviews, this 
tool can help prioritize and plan the sub-system review's. 



NASA Technical Report 


Technical Engineering Peer Reviews 

NASA 



Mission Metrics 
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Mission Metrics 
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Figure 7: System Review QFD 


System Review QFD is a variation of the traditional Quality Function 
Deployment. However, in the modified House I, the Project or Mission 
Requirements and Risks are mapped against Project or Mission Metrics. In 
House II, the Metrics are mapped against the different sub-systems sub-teams for 
the group. By weighting the requirements and correlating the metrics and sub- 
systems using standard QFD weightings, rough allocations of the project time can 
be estimated in a transparent and documented manner. This allows the project 
manager to better plan and dedicate resources according to these weights for the 
project reviews to ensure that the key risks are covered. Depending on the size of 
the systems or sub-systems, this same process can be used within a given sub- 
system to identify how the review should allocate coverage within one sub- 
system’s peer review. 

There are also tools which can be used on the sub-system level to help analyze the 
each sub-system, including FMEA. In contrast to the System Review QFD, these 
tools can be applied separately at each engineering review meeting. If applied, 
they can help the individuals involved at each meeting identify specific technical 
areas to discuss. 


Subsystem Level 

Because it is already used at many organizations, including at NASA, FMEA is 
perhaps the simplest and easiest method to identify problem areas in system. 
Failure Modes and Effects Analysis (FMEA) is a design tool that identifies, 
prioritizes, and alleviates potential problems in a given product, system, or 
process. Along with structured methods like Quality Function Deployment 
(QFD), FMEA is a risk management tool that provides decision guidelines to aid 
the team of designers to achieve a design that provides the most in terms of cost 
and quality. FMEA begins with the identification of the functions or 
requirements of the system and ways that they can fail to be satisfied. From that, 
the analysis identifies potential causes. Table headings are shown in Table 9 
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Table 9: Headings for a typical FMEA analysis 


Function or 
Requirement 


Potential Failure 
Mooes 


Potential Causes 
Of Failure 


Local Effects 


ErvtS Effects on 
Product, User, 
Other Systems 




© 
> . 
© 
to 


Detection Metnod' 
Current Controls 


I * 

8 P 

I N 


i Actions Recommended 
to Reduce RPN 


Responsibility and 
Target Completion 
Date 


For each failure mode and cause, the probability that they can occur must be 
identified and scored on a scale from 0-10. After the local and end effects are 
identified, the severity of each end effect must be scored on a similar scale. 
Finally, the detection rating scored refers to the likeliness of catching the failure 
modes before they happen. The product of these three terms is known as the Risk 
Priority Number (RPN) which gives a relative magnitude for each failure mode. 
The higher magnitude RPN’s should be a higher priority within the peer reviews, 
and the review team can discuss actions to reduce the RPN. 

If teams perform an FMEA analysis before each peer review, the peer reviewers 
can more quickly see problem areas already identified by the teams and either 
supplement the analysis with additional failure modes or investigate and discuss 
the high RPN items with the teams to devise corrective actions. The FMEA 
provides not only a systematic and consistent tool which forces the teams do to 
their homework before these reviews, but it also helps the peer reviewers 
understand the project and problems better. 

For analysis of the engineering tasks of each sub-system team, a process analysis 
such as design process FMEA (Chao 2003) can be applied. The goal of design 
process FMEA (dpFMEA) is to identify and apply rough weights to potential 
design errors as part of an error-proofing effort. Simplicity and speed are the goal 
of this analysis. A small group should analyze a design process in several hours 
rather than several days. Design process FMEA fully allows and expects changes 
to adapt to different processes. The design group performing the dpFMEA 
analysis should do the analysis using a spreadsheet environment to electronically 
record progress. In addition, this allows resorting of data to facilitate analysis. 

For each task, one should look through each category of the design errors. Start 
with each category and set of questions, and generate a list of errors, if applicable. 
If there is an error of the given type for a task, complete it on the FMEA table in 
the second column. For each error that is brainstormed, the immediate effects of 
the error should also be listed in Table 10. This approach looks at design errors 
from several key categories of error factors. (Chao 2003). 
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T able 1 0: Design error classification system categories 


Category 

Description 

Knowledge 

Inexperience or misunderstanding of the system 

Analysis 

Inaccurate assessment of the system 

Communication 

Mistransfer or misinterpretation of information 

Execution 

Improper or inaccurate implementation 

Change 

Unanticipated variation or modification 

Organization 

System or managerial deficiencies 


Quantifying the errors identified in either FMEA analysis identifies the areas of 
highest risk are often where the greatest opportunities are to be found. The issue 
is that it is difficult to quantify these different errors or risks. Expected cost is a 
good and easy to understand measure, but it often takes a full-time person much 
time to get good estimates on a dollar amount and probability for the different 
errors or risks. Tools in industrial engineering to estimate rework cost as well as 
surveys on reputation cost exist, but an engineer still cannot easily comprehend 
the eventual end effects of latent errors early in the life-cycle, much less measure 
them. One error can have multiple consequences of varying degrees of magnitude 


An alternative is to use the importance values calculated through Quality Function 
Deployment, based on importance to the customer requirements. This QFD 
method can be used to quantify the errors identified in either the traditional of 
design process FMEA analyses. The application of Design Error QFD, 
illustrated in Figure 8, relates customer requirements and risks to the design tasks 
as an alternate method of defining severity. (Chao 2003) 



Engineering 

Metrics 

Customer 

Requirements 

I 



Engineering 

Metrics 

Design Errors 

II 


Figure 8: Using QFD to rank design errors or risks 


Rather than mapping engineering metrics to parts in House II, the design errors 
are correlated to the customer requirements through engineering metrics. This 
quantifies the errors according to the customer values, which is important in 
determining quality of the design. These QED values give priorities for errors to 
consider from an error-proofing perspective. 


4.4. Approach 
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4.4.1. NASA lessons 


The nature of the human interactions that connect reviewers and reviewees can 
influence the results of the review as much as the technical understanding of both 
sides. Some design reviews are extremely adversarial and are even compared to 
prosecutors cross-examining the defendants in a legal setting. Others emphasize 
consensus building in a positive manner, so much so that people call them “group 
hug” reviews. Both approaches have their positives and negatives. The group 
approach tends to have broader and more even coverage of all the review topics. 
The more adversarial approach tends to get very detailed coverage, but only of 
one or two areas, usually the areas where the reviewer is most comfortable or 
familiar in. The project manager must be aware of these factors in the selection of 
the review board. 


One project manager has done much work on the psychology of risk and reminds 
reviewers to make corrections for personalities. If a person is optimistic in nature, 
his cost estimate will be optimistic as well. If the person is late to meetings, his 
schedule estimate will likely be underestimated. If the person has never worked 
on a badly overrun project, the next one likely will be. 


Another question is what the nature of the reviews should be. At Goddard Space 
Flight Center, the Management of Government Safety and Mission Assurance 
defines a spectrum of activities, from oversight to insight, to determine the 
adequacy of a product or process. Oversight typically entails onsite, in-line 
involvement with the supplier's processes and generally includes detailed 
monitoring of the process itself. In contrast, insight typically entails monitoring a 
minimum set of product or process data to provide an adequate understanding of 
the product or process. In general, the tendency as NASA is to lean more towards 
oversight if the risk is high and more toward insight if the risks are low. The 
strategy can change as the program progresses and more risk information 
identifies where changes are necessary or beneficial to reflect either an increase or 
decrease in risk. Within an area there may be varying levels of insight and 
oversight applied to portions of the surveillance. 


4.4.2. Literature review 

In light of recent events such as. the difficulty with the Mars missions and most 
notably the Columbia disaster, NASA as an organization has taken a deep look 
into changing the atmosphere around reporting problems. Former flight manager 
and a member of Columbia’s mission management team says it is important to 
note that “I wouldn’t look at this case as being all of NASA was wrong except 
one guy who had the answer. There has to be a more fundamental structural 
problem with how the communication broke down here.” Former astronaut Sally 
Ride has commented on the design review process saying that “This is a very 
personality-dependent thing, and these large meetings can be intimidating.” [7] 
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NASA chief Sean O’Keefe has promised dramatic change towards creating an 
atmosphere in which “we’re all encouraged to raise our hand and say something’s 
not right or something doesn’t look straight.” He has proposed changes such as 
going to a NASA web site to file anything anyone sees as being wrong, making it 
easy for anybody to participate and voice their concerns anonymously if they 
want. NASA is already well-known for its safety-reporting hotline and printed 
forms. [7] 

Domer (1996) researched characteristics of good and bad problem solvers. Good 
problem solvers favor expressions that take circumstances and exceptions into 
account, stress main points but don’t ignore subordinate ones, and suggest 
possibilities. They tend to use more qualified expressions like: now and then, in 
general, sometimes, ordinarily, often, specifically, perhaps, may, can, and be in a 
position to. Bad problem solvers tended to use unqualified expressions like: every 
time, absolutely, completely, nothing, only, must , and have to. 


Freedman and Weinberg (1990) suggested several rules of thumbs with running 
reviews. They recommend 2-10% of labor allocation should go to technical 
reviews. Reviews should try to run for about 2 hours. Starting at 10:00AM tends 
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produce rushed and superficial reviews as people continually look to the clock, 
hoping to get home early. 


4.4.3. Discussion 

The approach of the design review board obviously is greatly correlated with its 
composition. The formality of the design review can influence this behavior. 
Even the seating arrangements have psychological impact in peer reviews. By 
putting the review in a round table format and not having people stand-up and 
present the material, it prevents the exchange from being too formal or adversarial 
with a prosecutor’s mentality. Not having direct superiors or any sort of 
personnel evaluators is important for this reason. The peers involved should be 
concerned only with the problem at hand and not with impressing anyone. 
Certainly it is difficult to create guidelines on how to construct a review team 
based on personality types. Nonetheless, it is something that should be 
considered in constructing the teams just as technical ability is. 

Another key is for the reviewees to not fear tough review panels. They should not 
worry about “passing” the peer reviews and rather welcome the most insightful, 
experienced bruisers. As the ST-6 program manager said “Bruises now prevent 
bleeding wounds later.” The reviewees should listen to the criticism and not 
worry about defending the project. Reviewers’ suggestions of potential problems 
are items that the team must be aware of. 
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Though reviews should be informal and the team should be comfortable being 
open in discussing their concerns, there are still rules needed to ensure 
consistency. These reviews should still be in formal in the sense of ensuring the 
attendance of the invited reviewers and documentation of requests. If the 
proceedings are too informal and reviewers come and go early or late, the entire 
process will be much less efficient. 

Another view of insight and oversight is not as a continuum but as two distinct 
approaches that are both necessary to better mitigate error. Insight is not only 
necessary when risk is low. Insight into the process and system are both 
necessary before oversight can be implemented. Design Process FMEA is a good 
way to understand the risks inherent in the process. In the review process, the 
formal system reviews aim to oversee the project, primarily from a programmatic 
standpoint. Peer review should precede and complement those reviews and be 
used to gain insight on technical risks. 


4.5. Information technology 

4.5.1. NASA lessons 

Information technology is an important part of NASA’s push towards making 
their space and technology missions "faster, better, and cheaper". Different 
projects require different review practices. Larger projects are by definition 
required to have a certain rigor and will have the resources like information 
systems with configuration control and integrated project management. Smaller 
projects tend to use more informal tools such as e-mails and applications like 
Microsoft Project or Outlook for project management. 

Most projects keep track of their own project information, but there are very few 
centralized information repositories which are accessible by parallel or future 
projects. This likely results in much repeated work. There are a number of 
systems available including DocuShare and Livelink in use at NASA. 

DocuShare is a collaborative workgroup solution to allow project personnel to 
share their documents with each other and with their industry partners and 
affiliates around the world. NASA has 23 DocuShare servers and a total of 1,700 
users, including project managers, engineers, administrative staff, and scientists. 
Livelink is a highly scalable collaborative application that delivers Web-based 
file sharing for users in remote locations. This allows users to access and share 
files with users outside of NASA through the use of an Internet connection. 

There are a number of other knowledge systems, reference databases, and 
software tools in use or in development at NASA. The Lessons Learned 
Information System ( h ttp ://! 1 i s . n asa. go v/i i i s/i i i s/i 1 i s . h tm 1 ) is a database of 
different lessons from NASA projects. The Engineering for Complex Systems 
group at NASA Ames is developing a Mishap and Anomaly information system. 
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There are also software tools like DDP (Defect Detection and Prevention) being 
developed at NASA JPL to help identify risks. The major issue with these 
systems is that not all engineers are aware of them or how to use them. It is 
important for project managers to inform their teams about the availability and 
benefits of such resources. Training material with demonstrative examples that 
help with such tools are also a good way for engineers to learn their benefits 
themselves. 


4.5.2. Literature review 

Information technology can play a key role in error mitigation for design. (Chao 
2003) Computer technologies allow simulations of nearly any complex situation 
of interest. The flexibility of computer scenarios allows examination of 
experimental processes that were previously observable only in isolated cases. 
Industry and academia have explored numerous information technology tools 
towards error-proofing. Automation played a major role in improving quality on 
the assembly line. Similarly, automating a development process is necessary as 
part of the transition towards an increasingly electronic environment. In addition 
to modernizing the process to improve contact and communication between 
reviewers and reviewees, technology supports what people already know 
(Friedman 1997). 

Liu et al. (2001) discussed numerous ways web-based peer review systems can 
function, including as: 

1. An information distribution channel and management center for 
assignment submission and peer review 

2. A media for peer interaction and knowledge construction 

3. A storage center for knowledge construction procedures 

Gehringer (2000) also found that sharing reviews on a webpage is not sufficient to 
stimulate give-and-take between reviewers and reviewees. The reviewees do not 
“poll” the page periodically to see when new comments are submitted. A push 
system where both reviewers and reviewees are notified, like via e-mail, is 
necessary to update both sides. 


4.5.3. Discussion 

Information is essential part of the design process and the importance of 
information technology is increasing because of the increasing use of computers 
in design and communication. Computers can aid error-prevention by automating 
processes to prevent errors or aiding analyses through stand-alone software or 
even Excel templates and forms. 
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At a minimum, every review project should have a “locker” to store their files so 
that it is sharable within the group and with reviewers. There are a number of 
systems available to choose from. Ideally, one w'ith configuration management 
capabilities should be used and would make the implementation of this simple and 
straightforw'ard. The locker should be divided up into the different subsystems, 
just as the team and the peer reviews are, and also directories for system issues 
and one for background materials. The reviewers should be given read 
permission of all the files. The system ideally should have project management 
capabilities w'hich can not only schedule the review's but link the reviewers with 
relevant files. The system also needs to “push” e-mail to the impacted parties 
when changes are made to the documents. 

As mentioned previously, as important as storing the review content is to store the 
names and contact information of the different project and review personnel. By 
listing and linking this information objectively, future projects and reviewers can 
find resources to guide their w'ork and learn from the accumulated knowledge of 
the entire organization. 

Due to the increasing complexity of the systems in these projects, it is becoming 
increasingly difficult to understand them. Computer tools can play a big role in 
visualizing and demonstrating concepts. The use of these tools should be brought 
into the review process as much as possible and allow the reviewers to be more 
“hands-on” in understanding the systems, rather than just showing static pictures 
in a presentation viewgraph. 


5. Conclusions 

5.1. Implementation 

Because engineering peer review's are usually technical pre-reviews of specific 
subsystems in preparation for the formal system reviews, most system related 
issues are addressed after the peer review's. Considering this, and the fact that the 
system level reviews often must spend a half day or more or the scheduled formal 
review’s time reviewing the background of the mission or spacecraft in 
preparation for the “actual review,” one solution is to do this background and 
early system level analysis before both the informal and formal reviews. This 
encourages advance preparation for the technical review. Weinberg and 
Freedman (1990) found that at least 80% of review failures can be traced to lack 
of advance preparation. 

The goal of strengthening the process is not to force methodism. In complex and 
dynamic situations, the most reasonable strategy is to plan a rough outline and 
delegate to subordinates. These subordinates need considerable independence but 
also a thorough understanding of the overall plan. If these subordinates don’t 
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work together within the context of an overall plan, the unpredictability of the 
independent agents will simply add to the complexity of the overall problem. 
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Figure 9: System review timeline 


At this system background meeting, the design team can present the mission 
background to the formal review board and other peer reviewers. At the same 
time, initial discussion of the mission requirements and scope of the subsystems 
should be discussed to plan out the number of and depth of the various subsystem 
peer reviews. Identification of key contacts can facilitate initial communications. 
This meeting should be short, ideally half a day or so. 

Definition of what is expected from the review process should also be done at the 
system background meeting. Customer Value Chain Analysis can be done to 
list the relationship and the identities of the different engineering teams and 
reviewers. The steps of the analysis are to list the pertinent parties involved in the 
project and to identify the relationship and flow of information (like designs, 
requirements, or specifications), resources (like money), and requests (like 
criticisms, complaints, requests for actions, regulatory influences, votes). The 
graph should be analyzed to see what the value proposition of each party is to 
understand how they will impact the process. 
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Figure 10: 


Project Priority Matrix: often depends on product maturity 


The identification of the project priorities can also help plan the process. The 
construction of a project priority matrix begins by identifying the constrained 
factor. This can come from a hard limit on time-to-launch or time-to-market, a 
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hard budget or cost target, or a minimum set of new functions and features. Next, 
the target to be optimized should be identified. For example, does the team want 
quicker time to market, to minimize cost, or to maximize the function/feature set. 
The remaining item has to be accepted and whatever level achieved has to be 
lived with. The discussion can be preliminary and allow for the initial building of 
the System Review QFD without completing the entire QFD to help plan out the 
later peer reviews and formal reviews. The design team should also select a 
representative to identify issues where the reviewers had questions regarding the 
background of the project. 

At the end of the system background meeting, the design team should put together 
a pamphlet which summarizes the project background and key parameters, 
updating it as necessary, until the formal reviews. This pamphlet can be used by 
the review board members during the meeting to refresh their memory on the 
background of the project without spending the time to go through it all once 
again for the entire group. 


5.2. Complexities of distributed projects 

This report has concentrated on items for project managers to consider in 
directing their review process. However, many distributed and more complex 
projects at NASA and other organizations have several levels of hierarchy and 
different customer value chains. At the levels of the chief engineers and project 
sponsorship, it is difficult for the managers to be as hands-on with the design staff 
especially when they are distributed organizationally and geographically. Often 
times, they will have many different systems to manage, and involve the use of 
several sub-contractors from outside the organization. 

NASA and the DLR, German Aerospace Center, are working together to create a 
Stratospheric Observatory For Infrared Astronomy (SOFIA). A Boeing 747SP 
aircraft modified by L-3 Communications Integrated Systems to accommodate a 
2.5 meter reflecting telescope, SOFIA will be the largest airborne observatory in 
the world, and will make observations that are impossible for even the largest and 
highest of ground-based telescopes. It will be based at NASA's Ames Research 
Center at Moffett Federal Airfield near Mountain View, California, and is 
expected to begin flying in the year 2004. [5] 
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Figure 1 1 : SOFIA - Artist’s conception 


NASA does not interface directly with some of these organizations as USRA 
subcontracts the majority of the work. A team of industry experts led by the 
Universities Space Research Association is developing and operating the 
observatory for. As Figure 12 shows, there are a number of organizations 
involved in this project, and NASA does not have direct control or oversight of 
many of them. In addition, SOFIA has several components including science 
instruments, where the principal investigators are at various universities. DLR 
must work with German organizations like Kayser-Threde and MAN. 
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Figure 1 2: Organizations involved in the SOFIA Project 


The organizational structure here is nearly as complicated as the technology 
involved. Though NASA has an oversight role, it does not directly manage the 
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projects. They act as a customer and can call up major reviews. It is up to the 
contractors to monitor their own process. Not even USRA is involved deeply on 
the peer review process. There are some informal meetings, but otherwise not 
much insight for “external” customers. It is a matter of personality for these 
organizations as to whether NASA is even invited to some of these reviews 
because NASA is seen as a customer. 

Even on the most macroscopic level, each of these organizations has very 
different cultures. Academic, industrial, and government agencies have varying 
practices. Even the project teams within one organization can have very different 
practices whether the team is mechanical, electrical, software, or etc. Aircraft and 
aerodynamics groups, though seemingly related, have very different cultures. 

At a certain point, it is necessary for NASA to push for some process 
requirements in the review process. Though system reviews like the PDR and 
CDR are a start, due to the fact that these reviews address more programmatic 
issues and are much shallower on technical aspects, it is necessary for NASA to 
formalize the technical analysis, and thereby, the peer review process, to some 
degree. 


With its position very high up on this organizational structure, NASA is able to 
play a key role in interfaces. For the most part, people working on their portion of 
this system don’t need to worry about the overall structure and where they fit into 
the system. When different organizations need to work together, Interface 
Control Documents (ICD’s) help identify system concerns. SOFIA had over 30 
identified documents to negotiate interfaces between the major pieces of the 
telescope and the infrastructure. Power, cooling, heat loads, pressure, interface, 
and motions were just a number of the functional areas involved. 

The groups work together in Interface Control Working Group Meetings to 
iron out the design concept on both sides. This process is highly iterative and 
some times a bit rushed as both sides want the other side to finish their part so that 
they themselves can work to those specifications. NASA facilitated these 
meetings even though it was a non-contractual relationship, but it is still driven by 
each side’s desire to proceed along their development. To some degree, it is not 
in these contractors interest to better define these relationships. Lack of definition 
can often be used an excuse when there is a failure to meet schedule. As these 
working group meetings are essentially peer reviews, NASA does have the 
capability to get involved in this lower level review process. 

Another direction NASA can go is by requiring contractors, and their 
subcontractors, to work within some organizational guidelines in these peer 
review's as part of the contract. At some aircraft engine companies, for example, 
when certain contractors have had major failure events or other quality issues, 
they are required to go through error-proofing awareness training (EPAT’s). In 
response to problems, the aircraft engine company will send a team which wall 
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educate the contractor in error-proofing methods and then work with the 
contractor to implement error-proofing methods into their process. Failure to do 
so will affect the relationship of the company with the contractor. Though it is 
easier to leverage such methods in response to failure events, this is still a reactive 
process and means that a failure did go through. Ideally, this type of training 
should be done upfront. Also, it does not necessarily need to have a scope of this 
magnitude. Many organizations including the Manufacturing Modeling 
Laboratory at Stanford University have developed this type of material. 
fhttp://mrnl.stanford.edu s ) 


5.3. Summary 

There is no question that the formal review process at NASA is strong. However, 
because of the scope of these projects and the limited time and resources of those 
reviews, they must often concentrate largely on programmatic issues and can only 
cover technical issues in a fairly shallow manner. It is the role of the engineering 
peer reviews to identify the specific issues that can impact a project. 
Unfortunately, peer reviews lack consistency in implementation and depend very 
strongly on the strength of the project and section leaders. 

Currently different projects can have very different review practices and 
requirements. Larger projects have some advantages in terms of having more 
experienced people, as well as mentors available for the inexperienced. Smaller 
projects do not have this luxury - the projects are often the training grounds for 
new engineers. In addition, larger projects are by definition required to have a 
certain rigor in areas like configuration control. As a result, there is also less 
flexibility in larger projects. Because the large projects have more resources and 
priority, the quality and number of both the engineers and reviewers are also 
usually higher on larger projects. Because the formal reviews are also more 
rigorous, these projects can benefit from peer reviews as a good preparation work 
and a pre-review. 

The design review process is a weak process which depends on the individuals 
involved, both manager and engineer, reviewer and reviewee. Still, many at 
NASA do not recommend more requirements and formalism necessarily as a way 
to improve the process. Because design processes can vary so greatly in terms of 
not only size and scope, but also domain and technology, it is not feasible or 
practical to create a universal checklist for all reviews. There is an inherent 
tradeoff between the specificity of items in the list with the burden of performing 
such a review. The key to improving the design review process is to treat it as an 
activity of insight and not just oversight. Structured design methods can play a 
key role in this understanding. These tools are not static and require insight. In 
addition, they help document and share the groups’ thoughts and concerns. 

Failure Modes and Effects Analysis can identify key areas of concern before or in 
the review's. Product definition tools like the Project Priority Matrix, Customer 
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Value Chain Analysis, and Quality Function Deployment help prioritize resources 
in reviews. 


The organization, composition, scope, and attitude of reviews all play a role in 
their success. Information technology can also aid design reviews. In the end, 


NASA must acknowledge what design reviews can uui 
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limitations of their engineers. One project manager said that the basic reasons for 
bad cost estimates and risk plans are inherent to human nature due to blind initial 
optimism, overestimating the completeness of knowledge, underestimating the 
peril of unknown, and the belief that the worst just cannot happen. 


For most organizations, design reviews are the only line of defense against errors 
in the design process. Even if they are applied universally, they are still an 
imperfect gauge susceptible to human errors and will always allow some 
problems through. Nonetheless, they will always be an essential part of any 
organization’s efforts to error-proofing the development process. The key to 
using design review as a part of the design error-proofing toolkit is to recognize 
their inherent weaknesses. Design reviews are a dangerous tool. They can give 
the users a false sense of security. 

Unlike other error-proofing solutions, design reviews constitute a system that is, 
for the most part, already in place at most organizations. Improving the process 
does not require large capital investments in technology or even a change in the 
process necessarily. Design reviews can be impacted immediately. The key is to 
make these reviews as robust and thorough as possible. That can only be done 
with good pre-work, gathering both strong reviewers and reviewees and being 
committed to reviewing consistently. 

It is also important to regard design reviews as the first line of defense against 
errors and failures. In one representation of the different levels of error-proofing 
against design errors (Chao 2003), design reviews would fit at best as level 3 in 
robustness (refer to Table 1 1). There are still further steps that can be done to 
prevent and mitigate errors from occurring. 


T able 1 1 : Robustness level of error-proofs 


Level 

Type 

Description 

5 

Prevention 

Eliminate the possibility of 
performing an erring action 

4 

Detection 

Detect the error immediately 
after being made 

3 

Inspection 

Source inspection of completed 
parts 

2 

Improvement 

Improvement to simplify the 
process 

1 

Aid 

Tool or methodology to aid 
design/engineering 


4S 
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improvements of an organization’s developments can be done at different levels. 
It is not uncommon for some organizations to deny problems and rationalize them 
without fixing them by saying they are a “one of a kind” occurrence. With 
resources like lessons learned database, many of NASA’s improvements are on 
the problem level and very reactive and specific. Ideally, an organization should 
aim to fix the process and the system. 

One way to improve the peer review process is to target project managers and 
create a guide on how to conduct reviews, beginning with a background on the 
psychology of reviews. That can be followed with a training class, probably no 
longer than 2 hours. The first half-hour session can be on the current state of 
design reviews, a second on peer review techniques, and a third would be a 
retroactive example. A general guide like these would likely be accepted at both 
the upper management and project management levels. 

Many hesitate to have the organization place additional requirements on the 
engineers and managers as there are a number of regulations and requirements 
already in place. Certainly the biggest fears are the threat of more bureaucracy 
and requirements that don’t help people get their work done. Obviously, the 
organization, the project, and the individual some times have differing priorities, 
but the overall goal of all involved should be the same. Each side will need to 
make some adaptations for all involved to succeed. 

The key to using design reviews effectively is to understand where human 
limitations in dealing with complexity of systems impact the ability to review. 
Confidence in the process is different than blind faith in it. By having a strong 
and consistent process, both the process and the system reviewed can be 
understood. An organization can be both less dependent and use more effectively 
the review process if it better understands its limitations. 
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Appendix 

Resources 

Resources related to this research can be found on under “ECS/M AIS/Review 
Methods.” 

h ttps doc us h are . aen . n as a. go v/ doc us h are/ 


Meeting Schedule 

Visits/Interviews: 

- Jet Propulsion Laboratory (June 19-20, 2003) 

Jim Rose, Dave Swenson, Nick Thomas 

- Jet Propulsion Laboratory ( July 15, 2003) 

Art Chmielewski, Steve Prusha, Steve Comford, Tom Fouser 

- Ames Research Center (July 30, 2003) 

Chris Wiltsee, Nans Kunz, Ramsey Melugin 

- Jet Propulsion Laboratory ( October 16, 2003) 

Steve Prusha, Steve Comford 


Reviews/Conferences : 

- Kepler Ground Segment Peer Review (June 26, 2003) 

NASA Ames Research Center 

Moffett Field, CA, USA 

■ http://www.kepler.arc.nasa.gov 

- Space Mission Challenges in Information Technology (July 13-17, 2003) 
Pasadena Convention Center 

Pasadena, CA, USA 

■ http://smc-it.jpl.nasa.gov/ 

- NPI Roundtable: Reliability Validation and Tune to Market ( July 24, 
2003) 

Stanford University 
Stanford, CA, USA 

■ http://inml.stanford.edu/ 

- International Research Roundtable: Design for Life-cycle Quality 
across the Supply Chain (September 11, 2003) 

Swiss Federal Institute of Technology at Lausanne 
Lausanne, Switzerland 

■ http://www.epfl.ch 
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Customer Value Chain Analysis 



Figure 1 3: CVCA on the transfer of information 
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CVCA ON THE TRANSFER OF MONEY AND RESOURCES 
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Figure 15: 


CVCA ON THE TRANSFER OF REQUESTS AND CRITICISMS 


Figure 1 6: CVCA on the transfer of guidance and instructions 










